Conventional image sensors, such as CMOS and CCD sensors, integrate all light that impinges on them during the exposure time. This provides sharp images of static objects, but results in spatial blur for objects that move while the shutter is open. Objects that are not in focus are also blurred. The so-called motion blur is proportional to the exposure time and object velocity. The former is particularly troublesome when a camera operates under low light level conditions. Under such circumstances, long exposure times are desired to attain sufficiently high signal-to-noise levels such that the dark areas of a scene can be imaged adequately. Consequently, many cameras suffer from a classic trade-off between motion blur and dynamic range. The exposure times need to be long to capture enough light, but need to be small so as to reduce motion blur. Within the framework of the invention the word camera comprises cameras for taking photographs as well as cameras for video purposes.
A camera and method of the type described in the first paragraph are known from an article by Nagahara et al “Flexible Depth of Field Photography”, H. Nagahara, S. Kuthirummal, C. Zhou, and S. K. Nayar, European Conference on Computer Vision (ECCV), October, 2008.
In Nagahara et al a camera for taking photographs is shown in which the distance between the sensor and a fixed focus lens is varied. The sensor is swept over a distance during the exposure time. The sweeping distance is arranged to sweep a range of scene depth ranges in order to increase the depth of field. The prior art camera disclosed in Nagahara et al reduces out-of-focus blur. To reduce the out-of-focus blur the sensor is swept along the optical axis to cover certain depth ranges.
The sweeping of the sensor provides for a compound image, in effect being a combination of a number of images at various focal depths. A point spread function (PSF) can be calculated. A point spread function is, in effect, the image a point of an object would make on the sensor. For an object completely in focus the point spread would be zero, and thus the PSF would be a Dirac function. The Fourier transform of this function would be a constant for all frequencies. For a point not in focus the PSF is a spread-out function, for an object in motion while the camera is fixed, the PSF would be spread out over a distance due to the motion. From the PSF one can calculate an inverse point spread function (IPSF). Deconvoluting the compound image allows a sharp image to be obtained and an increased depth of field is obtained. This is due to the fact that, as Nagahara shows, when the sensor is swept the PSF for static objects at various distances becomes to a considerable degree the same. Thus, deconvolution the original image with one and the same IPSF would allow a sharp image at all distances, or at least an increased range of distance and the an increased depth of field is obtained for static objects.
Although out-of-focus blur and the reduction thereof may be and is important, a major problem, as explained above, exists and remains for moving objects, namely the motion blur, especially for larger exposure times.
Nagahara already mentions the problems associated with motion blur without giving a solution.
A known solution for reducing motion blur is to move the sensor perpendicular to the optical axis. This solution is known for instance from an article by Levin et al. “Motion-Invariant Photography”, A. Levin, P. Sand, T. S. Cho, F. Durand, W. T. Freeman. SIGGRAPH, ACM Transactions on Graphics, August 2008. In essence, this solution amounts to moving the sensor from left to right (or vice versa) during the exposure to reduce motion blur due to a horizontal motion.
Apart from the solution suggested in Levin et al to, motion blur can be inverted by means of video processing. This is achieved by motion estimation and inverse filtering along the motion trajectory. This is known for instance from U.S. Pat. No. 6,930,676. In practice however, the results of such a procedure suffer from inaccurate motion vectors, particularly for occlusion areas. One has to know the motion trajectory and deduce motion vectors from them to be able to do the inverse filtering. In many stand-alone cameras used in professional applications, motion vectors may not be available at all. For example, the recordings of many cameras used for surveillance or activity monitoring merely provide input to computer-vision-based analysis procedures (e.g., automatic detection of suspicious objects, fall-detection for elderly, etc). In these scenarios, the quality of the raw input frames is a determining factor for the performance of the detection system. Sufficiently accurate motion vectors may not be available on-the-fly within the camera and post-processing of recorded video is not an option in real-time monitoring systems. For a camera that takes a single snapshot it is fundamentally impossible to accurately determine motion vectors. At occlusion areas estimation of motion is also extremely difficult and inaccurate, if at all possible. At low light conditions the problems increase, due to the lack of light.
Second, most traditional cameras feature an adjustable shutter and aperture that windows the light coming through the lens in the temporal and spatial dimensions. These can typically be characterized as box filters (i.e. a constant sensitivity over a finite interval), corresponding to a sinc modulation in the corresponding temporal and spatial frequency domains. As a result, some high frequencies are fully suppressed during acquisition and cannot be recovered during inverse FIR filtering even when perfect motion information would be available. In practice, inverse filtering should be done with utmost care to prevent the amplification of noise and the introduction of artefacts.
In short, an effective and simple means for reducing motion blur is not known from prior art.